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PREFACE 


We  would  like  to  sincerely  thank  the  many  people  who  assisted  in  the 
conduct  of  this  research.  Pat  Kyllonen  and  Bill  Alley  provided  us  with  valuable 
comments  and  suggestions  on  this  article. 

The  research  reported  in  this  paper  was  conducted  as  part  of  the 
Armstrong  Laboratory,  Human  Resources  Directorate-Learning  Abilities 
Measurement  Program  (LAMP).  This  study  represents  basic  research  funded 
by  the  Air  Force  Office  of  Scientific  Research.  The  opinions  expressed  in  this 
article  are  those  of  the  authors  and  do  not  necessarily  reflect  those  of  the 
U.S.  Air  Force. 

Correspondence  concerning  this  paper  should  be  addressed  to  Valerie  J. 
Shute,  Armstrong  Laboratory,  Human  Resources  Directorate,  Individual 
Attributes  Branch,  Brooks  Air  Force  Base,  Texas  78235-5000. 


IF  PRACTICE  MAKES  PERFECT, 
WHAT  DOES  LESS  PRACTICE  MAKE? 


SUMMARY 

“Practice  makes  perfect'-more  time  spent  exercising  new  cognitive  skills 
results  in  improved  performance  and  reduced  cognitive  load  (e.g.,  Ackerman, 
1988;  Anderson,  1987;  Schneider  &  Shiffrin,  1977).  In  addition,  when  the 
number  (or  variety)  of  example  problems  is  small,  learning  tends  to  be  rapid, 
but  transfer  tends  to  be  weak  (e.g.,  Carlson  &  Yaure,  1990;  Gick  &  Holyoak, 
1987).  This  paper  examined  these  issues  in  a  controlled  setting  using  an 
intelligent  tutoring  system  teaching  novel  knowledge  and  skills  (i.e.,  flight 
engineering).  We  manipulated  the  tutor  to  yield  two  contrasting  learning 
environments:  “extended*  (12  problems  per  problem  set)  and  “constrained”  (3 
problems  per  problem  set).  These  environments  differed  only  in  the  number 
of  practice  problems  requiring  solution  in  the  various  problem  sets.  While 
subjects  in  the  shorter  version  were  expected  to  complete  the  curriculum  faster, 
we  wanted  to  examine  practice  effects  on  learning  outcome  (transfer)  and  skill 
acquisition  (error  and  latency  analyses).  Results  showed  that  while  subjects 
in  the  constrained  environment  completed  the  curriculum  significantly  faster  than 
subjects  in  the  extended  version,  there  were  no  differences  on  any  of  the 
outcome  tests.  Moreover,  when  the  data  were  examined  across  problem  sets, 
latency  and  error-type  differences  between  the  two  groups  were  found. 


INTRODUCTION 

An  enduring  expression  that  has  a  lot  of  empirical  support  in  cognitive 
psychology  is  that  “practice  makes  perfect”  (e.g.,  Ackerman, .  1988;  Anderson, 
1987;  Fisk,  in  press;  Schneider  &  Shiffrin,  1977;  Woltz,  1988).  Increasing  the 
time  allocated  to  exercising  a  new  cognitive  skill  typically  results  In  improved 
performance  and  a  reduced  cognitive  load.  A  related  finding  is  that  when  the 
number  (or  variety)  of  example  problems  is  small,  learning  tends  to  be  rapid, 
but  transfer  tends  to  be  weak  (e.g.,  Carlson  &  Yaure,  1990;  Gick  &  Holyoak, 
1987).  Thus,  more  and  varied  practice  problems  lead  to  rhore  effective  skill 
acquisition.  But  how  many  practice  problems  are  sufficient?  Alternatively,  how 
few  practice  problems  can  be  provided  that  will  yield  comparable  learning 
outcomes  to  richer  practice  environments? 

The  purpose  of  this  paper  is  to  investigate  the  effects  of  practice  on  learning 
outcome.  Other  questions  address  the  relationship  between  practice  effects  and 
the  learning  process.  In  particular,  what  are  the  effects  of  practice  on  learning 
curves  and  errors  made  during  skill  acquisition?  For  example,  do  subjects 
tend  to  compensate  for  fewer  practice  problems  by  adjusting  (increasing)  their 
time  spent  learning  each  problem?  Is  there  an  effect  of  practice  on  the  number 
or  nature  of  errors  made  during  the  learning  process? 
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We  will  present  results  from  a  large-scale  experiment  that  was  designed  to 
test  practice  effects  from  a  systematically  altered  tutoring  system  teaching  flight 
engineering  skills.  We  manipulated  the  tutor  to  yield  two  contrasting  learning 
environments,  differing  only  in  the  number  of  problems  the  learner  needed  to 
solve  in  each  of  nine  different  problem  sets.  The  version  with  many  problems 
was  called  “extended”  (12  problems  per  problem  set).  The  version  with  fewer 
problems  to  solve  was  called  “constrained"  (3  problems  per  set).  A  fixed  ratio 
of  4:1  existed  between  the  two  tutor  versions. 

The  simple  hypotheses  were  that  (a)  subjects  assigned  to  the  constrained 
environment  should  take  less  time  to  complete  the  tutor  because  there  were 
considerably  fewer  problems  for  them  to  solve.  However,  (b)  these  same 
subjects  were  not  expected  to  perform  as  well  on  the  posttests  compared  with 
subjects  learning  from  the  extended  environment  who  would  have  received 
considerably  more  practice  solving  tutor-related  problems.  In  addition,  we 

expected  that  (c)  subjects  would  tend  to  adjust  their  learning  times  per  problem 
where  those  in  the  short  version  would,  over  time,  take  longer  to  solve  each 
of  their  three  problems  (per  problem  set)  compared  to  subjects  in  the  longer 
version.  This  adjustment  was  expected  to  occur  gradually,  as  learners  realized 
they  had  fewer  problems  and  thus  invested  more  time  per  problem.  Finally, 
we  hypothesized  that  (d)  subjects  in  the  constrained  version  would  manifest 
more  conceptual  (rather  than  computational)  errors  during  learning  given  their 
sparse  practice  environment. 


METHOD 


Subjects 

The  subjects  in  this  study  consisted  of  356  males  and  females  participating 
in  a  seven-day  study  on  the  acquisition  of  flight  engineering  knowledge  and 
skills  from  an  intelligent  tutoring  system.  The  gender  distribution  in  the  sample 
was  approximately  75  percent  males  and  25  percent  females.  All  subjects 
were  high  school  graduates  with  a  mean  age  of  22  years.  Subjects  were 
obtained  from  a  local  temporary  employment  agency  and  were  paid  for  their 
participation,  consisting  of  forty-five  hours  of  testing  and  learning.  None  of  the 
subjects  had  any  prior  experience  or  training  as  flight  engineers  or  pilots. 


Flight  Engineering  Tutor 

The  tutor  was  originally  developed  at  the  University  of  Pittsburgh  (Lesgold, 
Bunzo,  &  Eastman,  1989)  and  then  modified  at  the  Armstrong  Laboratory  to  fit 
experimental  objectives.  The  tutor  was  designed  to  teach  knowledge  and  skills 
associated  with  a  flight  engineer’s  job.  Job  components  include  collecting  and 
analyzing  information  about  a  pending  flight  and  deciding  whether  various  factors 
(e.g.,  weather  and  runway  conditions,  type  of  aircraft)  indicate  a  safe  flight. 
There  were  two  parts  to  the  curriculum:  graph  reading,  and  computing  values 
for  inclusion  in  the  TOLD  (Take  Off  and  Landing  Data)  sheet.  The  focus  of 
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this  paper  is  on  performance  within  the  second,  more  substantive,  part  of  the 
tutor  dealing  with  completing  the  TOLD  worksheet. 

The  TOLD  worksheet  part  of  the  curriculum  consisted  of  nine  problem  sets 
and  involved  applying  the  graph  reading  skills  taught  earlier  to  fill  in  the  TOLD 
worksheet.  The  on-line  worksheet  was  designed  to  match  the  actual  forms 
used  by  flight  engineers.  Information  must  be  assembled  and  entered  into  the 
proper  cells.  Some  information  was  given  to  the  learners  (e.g.,  gross  weight 
of  the  aircraft,  wind  direction  and  velocity,  the  length,  heading,  and  conditions 
of  the  runway,  obstacles  in  the  flight  path).  Other  information  had  to  be 
derived  from  complex  graphs  and  then  entered  into  the  correct  cells  of  the 
worksheet.  For  instance,  the  wind-components  chart  is  used  to  compute  the 
headwind,  tailwind,  or  crosswind  components,  and  consists  of  two  superimposed 
charts:  polar  and  Cartesian  coordinates.  On-line  tools  were  available  to  assist 
the  learner  in  making  the  necessary  computations.  Some  of  the  tools  that 
could  be  used  with  the  wind-components  chart  included:  draw  vertical  or 
horizontal  lines,  add  a  radius  or  vector,  erase  lines,  redisplay  graph,  and  so 
forth.  A  “Help"  window  allowed  the  learner  to  read  related  information  about 
the  topic  under  study.  Learning  was  self-paced. 

The  first  (and  easiest)  set  of  problems  (sets  1,  2,  and  3)  involved  computing 
the  maximum  allowable  crosswind  given  the  gross  weight  of  the  aircraft  and 
the  runway  condition  reading  (RCR).  These  problem  sets  became  progressively 
more  difficult.  Prior  to  the  problem-solving  phase  in  each  problem  set,  concepts 
or  procedures  related  to  the  problem  set  were  discussed  and  illustrated  by  the 
tutor  in  detail.  For  example,  the  tutor  explicitly  demonstrated  how  to  figure 
out  the  maximum  allowable  crosswind  before  presenting  any  problems  (see 
Figure  1).  First,  the  relevant  chart  was  displayed  on  the  computer  screen. 
Then  the  computer  drew  a  vertical  line  up  from  the  x-axis  (corresponding  to 
the  given  gross  weight  of  the  aircraft)  to  intersect  the  appropriate  RCR.  A 
second  line  was  displayed  moving  horizontally  across  from  the  RCR  curve  to 
the  y-axis.  The  Intersection  at  the  y-axis  yielded  the  crosswind  value.  Text 
always  accompanied  the  visual  (dynamic)  displays,  for  example,  ‘The  MAXIMUM 
ALLOWABLE  CROSSWIND  FOR  TAKEOFF  chart  is  shown  below.  To  use  the 
chart,  values  from  two  variables  are  needed:  (1)  Gross  weight  of  the  aircraft 
and  (2)  Runway  condition  reading  (RCR).  RCR  is  a  measure  of  the  tire-to-runway 
coefficient  of  friction...* 

The  second  cluster  of  problem  sets  (4,  5,  and  6)  Involved  computing  the 
headwind,  crosswind,  and  tailwind  components,  respectively.  Here,  subjects 
had  to  apply  rules  learned  earlier  about  what  constituted  various  wind  types. 
For  example,  the  headwind  rule  was:  If  the  relative  wind  direction  is  less  than 
90  degrees  (i.e.,  0  to  89),  or  greater  than  270  degrees  (i.e.,  271  to  360),  then 
it  is  a  “headwind."  Figure  2  illustrates  a  “headwind"  problem.  Given  the  wind 
direction,  runway  heading,  wind  velocity,  and  gust  (see  Figure  2),  the  “relative 
wind  direction"  was  determined  from  subtracting  the  wind  direction  from  the 
runway  heading  (smaller  value  from  the  larger).  Another  rule  is  that  gust  values 
are  not  added  to  headwinds,  but  are  added  to  crosswinds  and  tailwinds.  A 
seven-step  procedure  is  shown  in  Figure  2  computing  the  headwind  and  crosswind 
components. 
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GROSS  WEIGHT  -  1000  LB 

Given  gross  weight  and  RCR  (runway  condition  reading), 
determine  the  maximum  allowable  crosswind  for  takeoff 
using  the  chart  above  and  the  tools  from  the  tool  menu. 


Figure  1 

Examples  of  Problem  Set  for  Determining  Maximum 
Allowable  Crosswind  Given  Gross  Weight  and  RCR 


The  final  group  of  problem  sets  (7,  8,  and  9)  required  the  learner  to  integrate 
all  earlier  problem  sets.  These  problems  involved  determining  the  maximum 
allowable  crosswind  values,  computing  the  wind  component  values,  then  deciding 
whether  to  proceed  or  abort  the  takeoff  based  on  the  computed  information. 

Learning  Outcome  Measures 

We  wanted  to  assess  the  acquisition  of  tutor-specific  skills  as  well  as  general 
graph  reading  and  interpreting  skills  that  may  have  been  enhanced  as  a  function 
of  learning  from  the  tutor.  To  achieve  this  end,  we  created  three  categories 
of  posttests:  (a)  Basic  graph  knowledge  and  skills,  (b)  Complex  graph  knowledge 
and  skills,  and  (c)  Tutor-specific  knowledge  and  skills.  Within  each  category, 
several  tests  measured  declarative  knowledge  and  procedural  skill  acquisition 
separately.  All  tests  were  administered  on-line  at  the  conclusion  of  the  tutor. 

1.  Basic  graph  knowledge  and  skills.  This  test  was  divided  into  five  parts 
and  consisted  of  50  items  altogether.  It  was  a  multiple  choice  format  (with 
10  alternative  responses  to  choose  from).  For  example:  The  horizontal  part 
of  a  graph  is  called  the  _ ?  (correct  answer  is  x-axis). 
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Headwind  Problem 

GIVEN: _ 

Wind  Dir.  =  200° 

Runway  Heading  =  235  ° 
WindVel.  =  15kts. 

Gust  =  5  kts. 


THUS: 

1.  Relative  Wind  Dir.  =  35° 

2.  Add  Vector  at  35  ° 

3.  Intersect  wind  vel.  (15  kts) 
and  wind  dir.  (35°)  - 

4.  HW  comp.  =  12  kts. 

5.  CW  =  Vel. -I- ^st  =  20  kts. 

6.  Plot  vert,  line  from  20  kts. 

and  35°  - 

7.  Crosswind  comp.  =  10.5  kts. 


Figure  2 

Seven>Step  Solution  for  Solving  Headwind  and  Crosswind 
Problems  from  the  Wind  Components  Chart 


2.  Complex  graph  knowledge  and  skills.  We  created  this  test  to  assess 
performance  on  more  complex  graph-related  problems.  There  were  three  main 
parts:  Understanding  functions,  Story  problems,  and  Complex  graph  interpretation. 
Posttest  2  items  were  designed  to  measure  graph  interpretation  skills  such  as 
understanding  functional  relations  and  multidimensional  graphs.  The  story 
problems  were  created  to  be  analogous  to  some  of  the  problems  encountered 
in  the  flight  engineering  tutor,  but  without  any  of  the  vernacular.  There  were 
30  items  in  this  test,  multiple  choice  format  with  six  alternatives  to  choose 
from. 

3.  Tutor-specific  knowledge  and  skills.  We  created  the  final  posttest  to 

assess  the  depth  and  breadth  of  knowledge  and  skills  acquired  from  the  tutor. 
All  three  parts  comprising  this  posttest  were  tutor-specific.  The  three  parts 
measured:  Declarative  knowledge.  Procedural  knowledge,  and  Graph 

interpretation.  There  were  45  items  altogether  in  this  posttest,  administered  in 
a  multiple  choice  format  with  six  alternatives  to  choose  from. 
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Error  Types 


Ten  different  types  of  errors  were  tallied  during  the  learning  process.  These 
were  classified  as  either  numerical  or  conceptual  errors.  Numerical  errors 
included:  (a)  incorrect  HW  or  TW  value  entered,  (b)  incorrect  maximum  allowable 
crosswind  value,  (c)  incorrect  crosswind  component  computed,  (d)  omitted  data 
in  TOLD  sheet,  and  (e)  failure  to  abort  an  unsafe  flight  (or  choosing  to  abort 
a  safe  flight)  due  to  incorrect  wind  component  value{s).  Conceptual  errors 
were:  (a)  reversing  axes  on  Wind  Components  chart,  (b)  reversing  axes  on 
Maximum  Allowable  Crosswind  chart,  (c)  reversing  vectors  on  Wind  Components 
chart,  (d)  interpreting  from  incorrect  curve  on  the  Maximum  Allowable  Crosswind 
chart,  and  (e)  using  an  incorrect  vector  on  the  Wind  Components  chart. 


Procedure 

Subjects  were  tested  in  oroups  of  approximately  20  persons,  and  there  were 
twenty  groups  tested,  total."  Each  p  oup  spent  seven  days  (about  six  hours 
per  day)  in  this  study.  Subjects  began  the  study  being  tested  on  basic  cognitive 
process  measures  (not  relevant  to  this  paper),  and  then  were  randomly  assigned 
to  one  of  the  learning  environments  of  the  tutor.  Directly  following  the  completion 
of  the  tutor,  subjects  were  administered  the  criterion  posttest  battery. 


RESULTS 

As  expected,  subjects  in  the  constrained  environment  did  complete  the  tutor 
faster  than  subjects  in  the  extended  environment  {Constrained  M  =  8.0  hr,  SD 
=  4.0,  N  =  181;  Extended  M  =  10.6  hr,  SD  =  4.5,  N  =  183).  An  ANOVA  on 
these  data  showed  the  difference  to  be  significant  (F  1.362  =  33.14;  p  <  ,001). 

But  a  more  interesting  question  addressed  how  well  the  respective  groups 
performed  on  the  outcome  measures.  Table  1  shows  each  of  the  individual 
(and  average)  posttest  scores  separated  by  the  two  versions  of  the  tutor. 
Even-odd  reliabilities  of  the  Posttests  were  high  (r  =  .91,  Posttest  1;  r  =  .91, 
Posttest  2;  r  =  .90,  Posttest  3). 

These  results  were  very  surprising.  In  all  of  the  outcome  measures,  there 
were  no  differences  between  the  two  practice  environments.  In  addition,  this 
finding  was  not  a  function  of  differential  incoming  knowledge.  A  pretest  was 
administered  to  all  subjects  before  starting  the  tutor  corresponding  to  the  same 
item  types  as  in  Posttest  1  (i.e.,  Basic  Graphs).  There  were  no  differences 


’Some  individuals  dropped  out  before  completing  all  parts  of  tbe  study. 


on  any  of  the  individual  tests  by  treatment  group,  or  in  the  overall  pretest 
score  {Pretest-Constrained  M  =  61.8,  SD  =  20.16,  N  =  181;  Pretest-Extended 
M  =  62.6,  SD  =  18.2,  N  =  183;  F  i.aea  =  0.19,  NS).^ 


Table  1.  Posttest  Scores  Separated  by  Learning  Environment 


Test 

Constrained 

(N=177) 

Extended 

(N=179) 

F 

Signif 

1  -  BASIC  GFIAPHS 


Graph  Knowledge 

60.40 

57.82 

0.81 

NS 

Read  Points 

53.45 

52.45 

0.17 

NS 

Read  Relations 

69.77 

66.15 

2.01 

NS 

Interpret  Points 

62.49 

63.80 

0.34 

NS 

Interpret  Relations 

52.26 

52.29 

0.00 

NS 

(Average) 

59.37 

58.48 

0.32 

NS 

COMPLEX  GRAPHS 

Functions 

27.82 

30.00 

0.35 

NS 

Story  Problems 

56.47 

59.32 

1.99 

NS 

Complex  Graphs 

69.44 

70.22 

0.10 

NS 

(Average) 

56.45 

58.37 

0.84 

NS 

TUTOR  SPECIFIC 

Declar.  Knowledge 

61.02 

60.89 

0.00 

NS 

Proced.  Knowledge 

48.08 

48.21 

0.00 

NS 

TOLD  sheet 

62.71 

65.86 

2.21 

NS 

(Average) 

59.12 

60.83 

0.65 

NS 

The  next  set  of  questions  concerned  differences  between  groups  as  a 
function  of  latencies  and  errors  per  problem  set.  Figure  3  shows  the  data  for 
the  three  groups  of  problems.  Problem  sets  1,  2,  and  3  all  involve  figuring 
out  maximum  allowable  crosswind;  sets  4,  5,  and  6  involve  computing  headwind, 
tailwind,  and  crosswind  values;  and  finally,  sets  7,  8,  and  9  involve  integrating 
the  prior  problem  sets.  The  first  row  of  graphs  represents  solution  time  by 
problem.  The  second  row  represents  error  data  (total  errors  per  problem). 


^In  addition,  oach  poattaat  was  regrassad  against  tha  maan  pretest  score  and  the  residuals  saved.  Comparing  outcomes  with 
these  reaiduaKzad  poattast  scores  again  showed  no  significant  differences  between  conditions. 
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Pfabl«nS6t«1,2,3(yu.Crouwtnd)  Prebtwn  Sol*  4. 5. 6  (HW,  TO,  XW)  Problem  Sot*  7, 8, 9  (Computing  TOLD) 


Figure  3 

Latency  and  Error  Data  Across  Problem  Sets 


For  the  easiest  problem  sets,  there  were  no  strong  differences  between  the 
two  groups  for  either  latency  or  error  data.  But  the  second  group  of  problem 
sets  shows  some  separation  between  learning  curves  whereby  subjects  in  the 
constrained  version  began  to  take  longer  per  problem  compared  to  those  in 
the  extended  version.  Finally,  for  the  complex  problems  (sets  7,  8,  and  9), 
the  gap  has  widened  considerably  for  the  time  data.  Subjects  appeared  to  have 
adjusted  their  learning  times  upward  in  the  constrained  version.  The  mean 
time  to  learn  these  problem  sets  for  subjects  in  the  constrained  version  was 
54.24  minutes  (SD  =  22.13,  N  =  177),  and  the  mean  time  for  subjects  in  the 
extended  version  was  35.77  minutes  (SD  =  15.34,  N  =  178).  This  finding 
represented  a  significant  difference  between  the  learning  environments  (F  1,353 
=  83.61,  p  <  .001). 
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In  terms  of  the  error  data,  only  “total  errors"  were  plotted.  Again,  the  same 
pattern  of  differences  between  learning  environments  emerged.  The  number 
of  errors  made  during  the  easiest  problems  (sets  1,  2,  and  3)  was  about  the 
same  for  the  two  learning  environments.  For  the  next  grouping  of  problems 
(sets  4,  5,  and  6),  a  slight  separation  between  groups  began  to  appear,  but 
this  difference  was  not  significant.  By  the  final  group  of  problems  (sets  7,  8, 
and  9),  subjects  in  the  constrained  environment  were  making  more  errors  than 
subjects  in  the  extended  environment.  This  difference  was  statistically  significant 
(F  1,353  =  25.68,  p  <  .001). 

To  determine  any  differences  In  the  types  of  errors  committed,  the  error 
data  were  separated  into  either  “numerical"  or  “conceptual"  errors  and  we 
computed  a  proportion:  numerical  divided  by  conceptual  errors.  To  test  for 
differences  between  the  two  practice  environments,  we  computed  a  one-way 
Analysis  of  Variance  (ANOVA)  on  this  proportion  by  environment.  Results 
showed  that  subjects  in  the  constrained  environment  had  a  higher  proportion 
of  numerical  to  conceptual  errors  (M  =  2.69;  SD  =  1.55;  N  =  181)  compared 
to  subjects  in  the  extended  environment  (M  =  2.12;  SD  =  0.88;  N  =  183). 
This  difference  was  significant  (F  1.352  =  18.82;  p  <  .001). 


DISCUSSION 

Practice  effects  were  investigated  in  relation  to  overall  learning  time,  learning 
outcome,  and  parameters  of  skill  acquisition  (latencies  and  errors).  The  first 
hypothesis  was  supported:  subjects  in  the  constrained  environment  did  require 
significantly  less  time  to  complete  the  tutor  compared  to  subjects  in  the  extended 
environment.  The  second  hypothesis,  that  subjects  in  the  extended  environment 
would  perform  better  on  outcome  measures  than  subjects  in  the  constrained 
environment,  was  not  supported  and  the  surprise  finding  was  that  there  were 
no  differences  on  any  of  the  outcome  measures  between  the  two  groups.  This 
finding  may,  in  part,  be  explained  by  the  third  finding  that  subjects  in  the 
constrained  environment  gradually  increased  learning  times  per  problem, 
presumably  to  compensate  for  having  so  few  practice  problems  to  solve  (thus, 
hypothesis  3  was  supported).  Finally,  it  was  hypothesized  that  subjects  in  the 
constrained  environment  would  make  more  conceptual  than  numerical  errors. 
The  opposite  was  found.  But  this  finding  can  actually  account  for  the  other 
unexpected  finding  about  the  comparability  of  outcome  measures  between  groups. 
Because  subjects  in  the  constrained  environment  were  investing  more  time  per 
problem,  especially  within  the  most  difficult  problem  sets,  they  were  able  to 
extract  the  conceptual  issues  from  the  problems,  but  it  cost  them  additional 
time  per  problem.  Thus,  the  errors  they  made  were  less  conceptual  and  more 
numerical  (i.e.,  a  higher  proportion  of  numerical/conceptual  errors  compared  to 
the  subjects  in  the  extended  environment).  If  they  had  made  more  conceptual 
errors  (in  relation  to  numerical  errors),  then  we  would  expect  to  see  differences 
in  outcome  performance. 
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In  conclusion,  less  practice  (1/4  the  problems)  can  actually  result  in  comparable 
learning  outcomes  to  more  extensive  practice  environments.  The  cost  is  in 
terms  of  more  time  invested  per  problem  as  well  as  less  precise  computations 
(i.e.,  numerical  errors).  But  numerical  errors  can  have  severe  consequences 
in  the  real  world.  Thus,  these  data  suggest  that  the  tutor  should  remediate 
numerical  errors,  which  it  currently  does  not.  Additional  research  is  indicated 
to  determine  the  boundary  conditions  under  which  we  can  see  these  same 
results.  In  other  words,  just  how  low  can  we  go?  Furthermore,  research  is 
planned  to  test  if  differences  in  retention  could  be  detected  between  the  two 
groups  after  a  significant  period  of  time  has  elapsed  (e.g.,  9  months  to  1  year). 
Findings  may  show  that  benefits  from  the  extended  condition  show  up  in  terms 
of  better  retention  compared  to  subjects  learning  from  the  constrained  environment. 
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